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. Defining the purpose of evaluation as "to provide 
data for deaision making^" a methodolcgy for educational evaluation 
v;as developed. The criteria of efficiency, completeness and focus (on 
the decision maker's priorities) were used* The eleinents of the 
methodology are: (1) negotiation of the contract j (2) design of the 
evaluation I and (3) implementation of the evaluation 
design—measurement, reporting the data, evaluation of the 
evaluation, redesign of evaluation^ Among the advantages of this 
methodology are that it suggests a practical approach to developing 
additional methodology and to revising existing methodology that does 
not work and that it provides criteria for evaluating its own 
effectiveness. (KM) 
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The purpose of this paper is twofold: (1) to provide an overview of 
the development of the Fortune/Hutchinson Evaluation Methodology and (2) 
to provide an overview of the current status of that methodology. Perhaps 
the best starting point is to address the question, "Why was this methodology 

developed"? Methodology was built in order to fulfill the specific purpose 

of providing data for decision making. 

Administrators, teachers, and other decision makers need (1) data for 

ongoing decision making, (2) data which they can really use and (3) evaluation 

procedures which do not Interfere with their accomplishing what they want 

to accomplish, 

"Evaluators need to raaet administrators' teachers' and other decision 
makers' needs for data for deelBion making. They need methodology that can 
be applied to people's needs for data that they can actually use in their 
deeision making processes. Evaluators also need methodology which avoids 
interfering with what those decision makers want to accpmplish. 

If evaluation methodologists are concerned with administrators' and 
evaluators' needs for useful data for decision making, they need to know 
what the criteria are for the accomplishment of that purpose and what 
operations will accomplish that purpose. 

The Fortune/Hutchinson Evaluation Methodology was deviloped using and 
reflecting methodological thinking, a logically deductive and empirical 
process for arriving at a complete, fully operational and systematic set 
of rules and procedures for accoinplishing a deflneable purpose. 

Purposes of Evaluation 

In order for any methodology to he developed, it Is first necessary 
to have a purpose, rhe concept of ''purpose'' is the key to methodological 
developmant. Second, it is necessary to realize that different purposes 
demand different msthodolpgies.^ 
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is the purpose, then a public relations methodology should be used. If 
the purpose is the production of universally genfirallEable knowledge, then 
a research methodology should be used, if recommending decisions to 
decision makers is the purpose, then program design or systems engineering 
mechodology should be used. The purpoia of the Fortune/Hutchinson Methodology 
is to provide data for decision making, and this purpose is distinguished 
from other purposes. 

Once a clear purpose has been established, it Is necessary to examine 
the implications of that purpose. One part of this process is to "test" the 
purpose against several criteria.- deslraability . operationallzahllity. and 
practicality. Operationalization of the purpose can be accomplished by 
the application of the Operationalization of Fuzzy Concepts methodology, 
which results In an operational specification of the purpose (Hutchinson and 
Benedict, 1970; Jones, 1971). The practicality of such a purpose, and of a 
methodology to fulfill that purpose Is both a logical problem and a field 
test problem. Finally, the deslreablllty critierla was easily met by a simple 
examination of the current status of evaluation in education. 

Current Stato of the Art 

The 1960's saw the topic of educational evaluation develop from the 
rather simplistic and narrow notion of evaluation as testing to a much . 
broader and larger content area within the still broader field of educational 
research. With the appearance in 1963 of Lee J. Cronbach's article, expandini 

^^O'^c^Pt oi evaluation, and even more so with the ■appearance in 1967 
of the AEEA Monograph Series on .Curriculum Evaluation (Tyler, 1967), 
educational researchers have becoma Increasingly and suddenly aware of the ■ 
great void in educational evaluation methodology. This void is fu 
. brought home to the resiarchers by the increasing demands; and requestr that : . 
Er!c- . : ■ . thalr desks from the li^d for evaluation a klUs in, dealing wl^ ; 



^ numerous funded projects, e.g. Title I, Title III and so on. 

At first, this void was merely elaborated upon within a very limited 
audience but with the continuing work of Tyler (1967), Stake (1967, 1969), 
and Stufflebeam (1967, 1969); the appearance of CIRCE at Illinois, the 
Ohio State Evaluation Center and the UCLA Evaluation Center among otherg; 
and most recently with the joint efforts of Phi Delta Kappa and AERA in the 
form of one of the most definitive works to date on the subject (Stufflebeam, 
et,al,, 1971), this audience has gron^n larger and larger. But, and perhaps 

more important, the shortcomings of the field have become more and more 
obvious. Despite the theoretical works of the above named group of outstanding 

educationists there still exist few evaluation methodologlsts or methodologies. 
In fact, we can sum up the state of the art as follows f 

(1) The area of educational evaluation theory and conceptualisa- 
tion is sadly lacking, 

(2) Now more than ever there is a need for comprehensive evalua^ 
tlon procedures to be developed, 

(3) To date, the devGlopment of procedures has not been done at 
a very rapid rate, 

mm Need for Methodological Research in Educational Evaluation 

Even a brief examination of the current state of educational evaluation 
methodologies makes it obvious that not only are there many gaps between 
theory and practice, but there are many gaps in the theories themselves. 
(Worthen, in a paper presented at this convention last year (1972) gave 
an excelient documentation and critique of the many, many inadequacies 
existing in current evaluation methodology,) Even such prestlgous ''evaluation 
theorists*^ as Scrlven. (1967) , Stufflebeam (1969, 1971), Stake (1967, 1969) 
and Glasi (1969) have documented in their various writings the great need 
for methodological research in educational evaluation today. This need ' 
- becomis even more obvious when one begins t inadequacies of the 

ErJC v "^^^^^^ models of evaluation put forward by these and other wrlteraV e.g/ C^ 
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EPIC, EPIE, Provus Discrepancy Model (Provus, 1969), and so on. 
IM Fortune/H utchinson Mccliodolngv of E ducational Evaluation 

In answer to this very obvioua and very immediate need of educational 
evaluation, namely the need for methodological development, Fortune, 
Hutchinson, and others set about doing such developmenc. Beginning with a 
more comprehensive arid more utilitarian definition of the puipose of evaulation. 
namely to provide. data for decision making, they" have prpdeeded to develop 
prescriptive, not merely descriptive, procedures for educational evaluation. 
In fact, they contend that the only legitimate function of this evaluation 
methodology is to provide data to decision makers for their decision making 
purposes. (It should be pointed out that they are not the first to use 
this concept in the field. The reader is referred to the 1963 article by 
Cronbach and the later work (1969) of Cronbach and Suppes , as well as the 
1969 article of Stuf f lebeam' s . ) Fortune and Hutchinson have, however, considered 
the concept to a further degree, and better incorporate the concept In their 
niethodology, than do others who seem to verbalize it more than build upon 
It, or even really seriously consider it (witness for example the recently 
published PDK-AERA monosraph, Stuf f lebeam, et.al,, 1971), 

Other implications of the purpose to provide data for decision making 
arose, linplicatlons. overlooked by other "decision-oriented' T.odels (Hutchinson, 
1972). Three "user" criteria Gmerged for evaluation practice: .(1) efficleneys 
An evaluation is efficient to the eKtent that it provides only that data 
which a decision maker actually uses ; (2) coK.pleteness : An 'evaluation is 
complete to the extent that it provides all tha data needad by a decision 
maker; and (3) focus: An evaluation is focused to: the extent It provides all 
the data for the decision makers highest priority needy. 

These three "user" criteria have counterparts on the eva.luntlon 
methodology level, I.e. "evaluator" criteria! . (l)efficle^ the evaluator 

«^°"t?""i"8^ h naker and : ; 
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continuing review by the decision mak^r; (2) completeness implies that the 
methodology Is tested for completaness as to the decision makers' needs 
on a continuing basis; and (3) focus implies that mechodology use decision 
maker priqritles at every stage, rather than the evaluator's or someone 

* 

else's. 

This levol of analysis of the implications of the purpose to provide 
data for decision making yields yet another level of criteria i tnat of the 
methodologists, or of methodological development. To reiterate an earlier 
PQlnt, the purpose is the key to methodological development, Evaluators 
need procedures whose effectiveness can be measured and which can be 
revised if they do not work. Field testing of pieces of the methodology ihould 
occur under simple, available conditions where identification of what 
doesn't work can occur, rather than in giant, complex studies, where confounding 
resulti abound. And one final implication is that methodology for evaluation 
will probably never be complete, so the methodologists work will never end. 
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Following an examination and dellnaation of the implications of such 
a purpose, the methodologists then proceeded with the development of the 
actual elements of the methodology. This section of the paper is an overview 
of the major conceptual elements of the evaluation methodology^ with some 
discussion of the purpose of each element, (Space does not allow a, complete 
delineation of prescriptive steps here. However, these will be made available 
as a handout,) 

1.0 Negotiation of the contracts 

1*1 Explication of the evaluation methodology and determination 
©f whether it satisfies the needs of the temporary decision 

maker, ■■ ■ r-- ---.^ . . - 
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This step provides for Idontlf icotlon of the temporary decision 
maker (the person controliing Che evaluation resources); a 
statement of the purpose and an overvlaw of the methodoiogy; 
and the securemenC of a committment from the temporary decision 
maker that this is what he wants. 

1.2 Identification of the enterprise. 

The enterpriSG is defined as that which is to be avaluated, 
or that area in which decisions are to be made on the. basis 
of information to be gathered. Here .the enterprise is delineated, 
including its purpose, scope, etc. ' 

1.3 Elimination of misunderstanding. 

Thii is done to insure a mutual understanding between evaluator 
and decision maker and to prevent the evaluation from being 
erroneously designed. 

1.4 Identification of resources for evaluation. 

The temporary decision maker identifies those resources of the 
enfterprlse available to devote to the evaluation. r.esources are 
of two major kinds: those to be divided for evaluation among the 
various decision makers of the enterprise and those to be divided 
among the various evaluation tasks for each decision maker. The 
scope of the evaluation is equal to the amount of resources availabl* 

1.5 Identification of decision maker(s). 

All antorprise's have more than a single decision maker (unless 
the enterprise is defined as a single Individual). A decision 
maker is defined as a person for whose decision making needs 
evaluative data are to be gathered. It is necessary and important 
to define and identify those decision makers, as well as their 
priority order, for each is a potential user of data and each 
potentially needs different data. The evaluator has to know which 
decision maker(s) he will have to operate with and in which order. 

1.6 Preparation of the .contract. 

The actual agreement on the scope of the evaluation is committed 
to writing here before the evaluation proceeds. 

2.0 Design of the evaluation 

2.1 Identification of goals for each decision maker. 

The evaluatbr allcits the goals or Intents of each decision maker 
for whom information will be gathered. These are tested for 
completeness and systematically ordered as a guide for proceeding 
with the evaluation. The purpose is to arrive at as complete an 
approximation as possible of goals/Intents of each decision maker 
as specified in the contract, 

2.2 Identification of parts of the enterprise for each decision maker. 

^Thls IS a systems analysis for evaluation from the perspective 
of each of the decision makers for whom data is to be provided 

Decision makers need data not only (or even usually)' abo 

global enterprise but rather about specific parts or aspects " 

of that enterprise. For data to be provided about parts rather than 

II L".done":Jer' ^ '^^'^ ^^^^^ empL^VLd ' 




'2.3 Matching of goals to parts for each decision maker. 

The goals arrived at above (2,1) are matched to the appropriate 
parts (arrived at in 2,2) in order chat ic be knowii which goals 
belong to which part or are held for each part. This is done 
to provide a more efficient evaluation design and to provide more 
useful data for decision making, 

Us Operatlonallzation of goals for each decision maker. 

Goals/intents are usually '^fuzgy'S i.e. global, vague, general. 
This process systematically takes each goal and has the decision 
maker break it down into its directly observable and measureable 
components. This is done by a technique called the Operational^ 
ization of Fuzzy Concept. These components are tea ted for completeness 
and then prioritized, 

2,5 Development of observational technlquGs. 

Observational techniques are designed for the first priority 
operatlonalized components of each decision maker's goals. Ideal 
criteria for observational techniques are that they be used 
directly, under natural conditions, unobtruilvely . If available 
techniques do not fit these criteria, unique techniques are designed 
for the component at hand. These techniques are matched with 
resources to insure that they are not too costly vis a*' vis 
. evaluation resources. (Mien this Is accomplished, the process Is 
recycled back for the next priority operatlanalized components 

3.0 Implementation of the evaluation design, 

3*1 Implementation of measurement. 

Data recording devices are developed for the observational 
techniques developed (2.5)* Sampling Is done, if appropriate, 
both of observational techniques and of the target population! 
Tb^n the actual observations are carried out. Data is reported 
(cf, below) and plans to repeat the observation are designed as 
appropriate. (Recycle back for the ne^tt priority operationalized 
components as resources permit.) 

3.2 Reporting the data. 

Data is reported (on the results of 3,1) to the approprlata 
decision makers from the list of decision makers and in an 
efficient and appropriate manner, i,e. relating back to the 
observational techniques used, the operatlonalized componentCs) 
they are used for, for which goal and which part, and for which 
priority decisionmaker, 

3.3 Evaluation of the evaluation. 

The ©valuator determinis the extent to which decliioni were made 
on the data provided. He determines the amount of data provided 
which was used in the decision making process. He determines if 
the data was provided in time for the needs of the decision maker 
and if the decision maker had more pressing needs for which 
data were not provided. ' 

3.4 Redesign of evaluation. 

Redesign is systematically planned for the whole process and 
-for each sub-process as determined by, or. asked for by., either 
" ■ V t:he temporary decision maker or the evaluator. - 

- It is first determined if redeslen , - ■ 

-; ^^"^ " necessary and then for which 



parts of the evaluation it is to be done. The redesigned 
part.<s) would then be tested and adopted or redesigned as 
appropriate. 

These various elements are in various stages of development. The 
entire methodology has been field tested at levels varying from a single 
Integrated day, K-1 program, up through evaluation of school wide programs 
Various elements^ e.g. '-.he goals process, are currently being further deve 
and field tested. The OFC process has not only been field tested formally, 
but has had an substantial amount of dissemination and implementation. 
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The existence and further development of F/H has some significant 
implications for methodological development, not only In evaluation but 
in other areas of applications in education and the social sciences. The 
development of F/H to this point in time suggests a feasible and practical 
approach to developing additional raethodology and to revising existing 

ethodology that doesn't work. There exists a methodological modification • 
process which is analagous to theory modification in research. Gaps In 
extant methodologies, as well as methodologies under development can be 
identified an.d filled. Finally, this process providefi the basis for both 
decision oriented k'.nd conclusion oriented research. 

F/H has Important Implications for evaluation practice clsoi First, 
it provides criteria for evaluating Its own rff ectlveness . It provides ' 
a practitioner with step-by-step specif ications , not just admonitions, 
and general rules of good procedures. An evaluator can have a clearly 
defined sphere of activity, which means that he can explain to persons who 
ask him to do something else. i.e. to accomplish another purpose, why' 
somethipg else is not done and is not appropriate. In using this methodology. 
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.■decision maker cooperation and commitment is essential. This implies that 
an evflluator will be spending more cime working with people and less time 
working with data. Finally, with a methodology, an evaluator can provide 
his clients with greater assurance that he is neutral, that he is not giving 
prejudiced data' and that he will not be making ,the enterprise's decisions. 

The existence and development of F/H has implications for users of 
evaluation also. Methodology of this nature retains for a user his ethical 
and legal authority for making decisions. Methodology of this nature 
assures a user the data will be as useful to him as possible given his 
cooperation, the evaluator 's skill and the appropriate resources. The 
methodology assures that data which the decision maker does not need or will 
not use, will not be collected. This methodology provides criteria by which 
alternative evaluation procedures can be judged. 
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